System and method for using pattern vectors for video and image coding and decoding

ABSTRACT

An exemplary embodiment of the invention relates to a method of using pattern vectors for image coding and decoding. The method comprises converting a block of image data into a set of transform coefficients, quantizing the transform coefficients such that a number of the coefficients become zero, constructing a single entity or bit vector indicating which coefficients are non-zero, coding the single entity or bit vector as an integer using an adaptive, semi-adaptive or non-adaptive arithmetic coder, coding the values of the coefficients in any fixed order, using an adaptive, semi-adaptive or non-adaptive arithmetic coder, or some other coder, and coding all coefficients except the zero coefficients. The system and method of decoding data relate to the corresponding hardware and process steps performed by the decoder when decoding a bitstream coded as described herein.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/288,280, filed May 27, 2014, (now U.S. Pat. No. 9,049,421), which isa continuation of Ser. No. 13/925,272, filed Jun. 24, 2013, now U.S.Pat. No. 8,755,616, which is a continuation of U.S. patent applicationSer. No. 12/252,767, filed Oct. 16, 2008, now U.S. Pat. No. 8,472,729,which is a continuation of U.S. patent application Ser. No. 11/684,841,filed Mar. 12, 2007, now U.S. Pat. No. 7,454,071, which is acontinuation of U.S. patent Ser. No. 10/086,102, filed Feb. 28, 2002,now U.S. Pat. No. 7,206,448, all of the above cited applications areincorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a system and method of videocompression coding and decoding and particularly to a system and methodof using pattern vectors for video and image coding and decoding thateliminates two-dimensional coding of transform coefficients and therequisite zigzag scan order or alternate scan order.

2. Discussion of Related Art

Transform coding is the heart of several industry standards for imageand video compression. Transform coding compresses image data byrepresenting samples of an original signal with an equal number oftransform coefficients. A sample of the original signal may be thesignal itself, or it may be the difference between the signal and apredicted value of the signal, the prediction being done by any of anumber of widely-known methods. Transform coding exploits the fact thatfor typical images a large amount of signal energy is concentrated in asmall number of coefficients. Then only the coefficients withsignificant energy need to be coded. The discrete cosine transform (DCT)is adopted in standards such as the Joint Photographers Expert Group(JPEG) image coding standard, Motion Picture Expert Group (MPEG) videocoding standards, ITU-T recommendations H.261 and H.263 for visualtelephony, and many other commercially available compression systemsbased on some variations of these standard transform coding schemes.

Transform coding is a block-based image compression technique in whichthe input image is partitioned into fixed-size small blocks and eachblock of pixels is coded independently. FIG. 1, FIG. 2, and FIG. 3illustrate a standard method of block-based image compression. As shownin FIG. 1, in a typical transform encoder, an input image (101) ispartitioned into blocks (102). The blocks are usually square but may beof any rectangular shape, or in fact may be of any shape at all. FIG. 2illustrates the data 202 in a block is transformed by a sequence oflinear operations in the encoder into a set of quantized transformcoefficients. A predictor 204 may predict the sample values in the blockto yield a predicted block 206. Many such predictors are known in theart. A difference operator 208 computes a difference block 210representing a difference between the image data 202 and the predictionblock 206. A transform operator 212 transforms the difference block 210,typically a discrete cosine transform (DCT), into a set of transformcoefficients 214.

If the input block is rectangular, the set of transform coefficientsform a rectangular array. The transform coefficients y_(k), 1≦k≦K, arequantized independently by distinct quantizers 216 to generate a set ofindices, referred to as the quantized transform coefficients 218.

FIG. 3 shows that the indices are converted by a predetermined scanorder 302, typically one that zigzags through the quantized transformcoefficients in increasing frequency order, to produce a list oftransform coefficients 304. The list of transform coefficients isrewritten as a set of (run, level) pairs 306. The “run” component ofeach pair is the count of the number of zero coefficients before thenext nonzero coefficient; the “level” component is the value of the nextnonzero coefficient. The (run, level) pairs are mapped by a codewordmapper 308 into a sequence of bits 310 that are output to the channel tobe transmitted to the decoder.

FIG. 4. shows part of an example mapping between (run, level) pairs 402and codewords 404. One codeword 406 is reserved to indicate that thereare no more nonzero coefficients in the block, i.e., to indicate theend-of-block condition 408.

As shown in FIGS. 2 and 3, the basic process for transform codingincludes the following steps: converting a block of image data into anarray of transform coefficients (214); quantizing the transformcoefficients such that all, some, or none of the coefficients becomezero; the zero coefficients are typically the high-frequencycoefficients (218); ordering the coefficients in a list according to afixed order, typically in a zigzag scan ranging over the coefficientsfrom low to high frequency in both the horizontal and verticaldimensions, so that the zero (high-frequency) coefficients tend to beclustered at the end of the list (302); coding the list of coefficientsas a sequence of (run, level) pairs (306); assigning a codeword to eachpair according to a code such as a Huffman code (308); and using asingle reserved codeword to signify the “end of block” condition, thatis, the condition that all nonzero coefficients in the block havealready been coded (406,408).

The run component of each pair is the length of a run of zerocoefficients in the coefficient ordering, and the level is the actualvalue of the next nonzero coefficient. Each possible (run, level) pairis mapped by a fixed, previously determined mapping to a codeword basedon a variable length prefix-free code (e.g., a Huffman code). Onecodeword 406 of the code is reserved for the “end-of-block” indicator408, meaning that there are no more nonzero coefficients in the block.

There are deficiencies in transform coding. The method requires carefultuning of the coder. The following entities need to be carefullydesigned and matched to each other: (1) the coefficient ordering; (2)the variable length code; and (3) the matching of (run, level) pairs andthe end-of-block condition to codewords. In addition, related codingschemes fail to take advantage of correlations between coefficientsother than those implied by the fixed coefficient ordering. Further, theuse of prefix-free codes means that some compression inefficiency isinevitable.

Next, this disclosure discusses arithmetic coding with reference to FIG.5 Arithmetic coding is a method of coding according to which a sequenceof events, each with its own probability distribution, is coded, eachevent using the smallest number of bits theoretically possible given theprobability of the event. This number of bits is not restricted to beingan integer. An arithmetic coder retains state information betweenevents, and makes use of this state information to allow coding multipleevents with a single bit, and to allow the coding for a single event toextend over one or more full or partial bits.

FIG. 5 illustrates an example arithmetic encoder. The encoder containsprobability distributions 501, 502, 503, . . . , 504 for all possibleevents that can occur in different contexts C₁, C₂, C₃, . . . , C_(N).An event 510 is input to the coder, along with its associated contextidentifier 520. A selector 530 selects one of the stored probabilitydistributions 532 based on the context identifier. The arithmeticentropy coder 540 transforms the event, the selected probabilitydistribution, and the internal state of the arithmetic coder 550 into asequence of bits 560 to be output to the channel for transmission to thedecoder. The internal state 550 and the selected probabilitydistribution are updated.

A theoretical arithmetic coder uses unlimited precision arithmetic, andis not practical. In the related art there are a number of “approximatearithmetic coders.” These are approximate in the sense that the numberof output bits is nearly theoretically optimal, but not exactly so. Theresult of coding and decoding is a complete and exact reconstruction ofthe original sequence of events; it is not “approximate” in any sense.The term “arithmetic coding” invariably refers to use of an approximatearithmetic coder.

Many approximate arithmetic coders are designed to code binary events,that is, events that can have one of only two possible values. It is atrivial and obvious use of a binary arithmetic coder to code non-binaryevents by decomposing the non-binary events into a sequence of binarydecisions, each coded as a binary event by a binary arithmetic coder.

What is needed in the art is an improvement image coding and decoding.

SUMMARY OF THE INVENTION

What is needed in the art is a transform coding technique and transformdecoding technique that do not have to be tuned for all possible imagesahead of time. An adaptive arithmetic coder handles that requirement.Further improved coding and decoding efficiency may be achieved byremoving the need for an end-of-block signal. These and other advantagesare provided according to the present invention.

An exemplary embodiment of the invention relates to a method of usingpattern vectors for image coding. A computer device performs the method,as will be understood by those of skill in the art. With reference toFIG. 6, the method comprises converting a block of image data into anarray of transform coefficients (602) and quantizing the transformcoefficients such that all, some, or none of the coefficients becomezero (604). The method for coding image data further comprisesconstructing a bit vector indicating which coefficients are non-zero(606), and coding the bit vector as an integer using an arithmetic coder(608). The step of coding the bit vector may be accomplished using anadaptive arithmetic coder, semi-adaptive arithmetic coder or anon-adaptive arithmetic coder. The computer device codes the values ofthe coefficients in any fixed order, using an adaptive, semi-adaptive,or non-adaptive arithmetic coder.

Another embodiment of the invention relates to a method of decoding abitstream coded according to the above-mentioned method. This embodimentcomprises a method of decoding coded data comprising: decoding a bitvector coded as an integer using an arithmetic decoder wherein thevalues of the transform coefficients are decoded in any fixed order,deconstructing the bit vector to determine which coefficients arenon-zero, dequantizing the non-zero transform coefficients, andconverting the dequantized transform coefficients into block image data.

The number of coefficients transformed to zero depends on the imageblock itself and the quantization step size. The coarser thequantization, that is, the larger the quantization step size, the morecoefficients become 0 and the worse the reconstructed image looks. Fortypical images, the energy compaction properties of the DCT often causethe high frequency coefficients to be smaller than the low frequencycoefficients. A typical image contains hundreds, or even thousands, ofblocks, and a typical video segment contains tens of images everysecond. Effective compression depends on the fact that, on average, manyof the transform coefficients for many of the blocks are zero, and canbe transmitted with very few bits. The essence of the present inventionis a new and better method of using a very small number of bits toindicate the zero coefficients.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing advantages of the present invention will be apparent fromthe following detailed description of several embodiments of theinvention with reference to the corresponding accompanying drawings, inwhich:

FIG. 1 illustrates an operation of dividing an image into a group ofblocks;

FIG. 2 illustrates a known sequence of operations in image and videocoding to convert one block of an image or video frame into an array ofquantized transform coefficients;

FIG. 3 illustrates a known method of converting an array of quantizedtransform coefficients for one block into part of a bitstream;

FIG. 4 illustrates an example of part of a mapping between (run, level)pairs and codewords from a prefix-free code;

FIG. 5 illustrates a known method for performing arithmetic encoding;

FIG. 6 illustrates a method of coding image data according to anembodiment of the present invention;

FIG. 7 illustrates an example method of coding zero transformcoefficients according to the present invention;

FIG. 8 illustrates an example method of coding nonzero transformcoefficients according to the present invention; and

FIG. 9 illustrates an example method of decoding a bitstream generatedaccording to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention may be understood with reference to FIG. 6. FIG. 6shows a sample method of using pattern vectors for image codingaccording to an aspect of the invention. The method may be performed bythe hardware components known to those of skill in the art. The methodcomprises converting a block of image data into an array of transformcoefficients (602) and quantizing the transform coefficients such thatall, some or none of the coefficients become zero (604). The methodfurther comprises constructing a bit vector indicating whichcoefficients are non-zero (606) and coding the bit vector as an integerusing an adaptive, semi-adaptive or non-adaptive arithmetic coder (608).Those of skill in the art will be aware of such arithmetic coders. Hereit is noted that although a bit vector is referenced, the core idea ofthe present invention does not necessarily require the use of a bitvector given that the invention's principle is that all the zero andnon-zero coefficient information is combined into a single entity forcoding. Any related data whose relationship is not clearly defined canbe coded according to the principles of the present invention.

For example, in another aspect of the invention, a method of coding datanot having a clearly defined relationship comprises converting the datainto transform coefficients, quantizing the transform coefficients suchthat all, some or none of the transform coefficients become zero,constructing a single entity from the quantized transform coefficients,and coding the single entity using an arithmetic coder wherein thevalues of the transform coefficients are coded in any fixed order. Oneexample of the single entity is the bit vector discussed herein, butother entities may also be used.

Next, the method comprises coding the values of the coefficients in anyfixed order, using an adaptive, semi-adaptive or non-adaptive arithmeticcoder, or any other coder (610). Each coefficient is coded according toits own context, possibly based on which coefficient it is and possiblybased on other factors. The method includes coding all coefficientsexcept the zero coefficients indicated above (612).

The novel steps of the invention are further illustrated in FIG. 7 andFIG. 8. FIG. 7 illustrates the construction and coding of the singleentity or bit vector. Conceptually, a computer device rewrites the arrayof transform coefficients 702 as a list of transform coefficients 704.The method according to the present invention will be operated on acomputer device having a processor operating a program of instructionsto perform the data coding operations disclosed herein. Any number ofcomputer devices may be used and such various computer devices are knownto those of skill in the art and thus not illustrated.

A bit vector 706 has the same number of bits as the number ofcoefficients in the transform coefficient list, and there is aone-to-one correspondence between coefficients in the coefficient listand bits in the single entity or bit vector. The bit vector thusrepresents a significance map for the one-dimensional list 704 oftransform coefficients. Setting each bit in the bit vector where thecorresponding coefficient in the coefficient list is zero fills the bitvector. The bit vector is then reinterpreted as an integer 708. Anarithmetic coder 710 encodes the integer 708, with the context beingidentified as the “bit vector” context 712. The arithmetic coder outputsbits to a bitstream 714. The arithmetic coder 710 is as described aboveand illustrated in FIG. 5.

The computer device codes the values of the nonzero coefficients in anyfixed order, using any coder. The coder may be an adaptive,semi-adaptive or non-adaptive arithmetic coder, or it may be any othercoder. Most arithmetic coders consist of both a probability estimationpart and an entropy coding part. The probability distribution estimatesfor all events may be fixed ahead of time for all users of the coder; anarithmetic coder with this property is called “non-adaptive.” Theprobability distribution estimates for all events may be computed beforea use of the coder, and transmitted to the decoder before codingcommences; this distribution is then used for the entire use of thecoder. An arithmetic coder with this property is called “semi-adaptive.”The probability distribution estimates that the coder uses may changefor some or all events during the use of the coder in such a way thatthe decoder can make the same changes to the probability distributionestimates. An arithmetic coder with this property is called “adaptive.”In an adaptive arithmetic coder, it is possible to initialize one ormore of the probability distribution estimates to some predeterminedvalues. This often leads to faster adaptation. A typical use of anadaptive arithmetic coder is to always initialize all probabilitydistributions to values that are typical for the type of data beingcoded, then during a given use of the coder to adapt the appropriatedistributions after each event is coded. If the coefficients are codedusing an arithmetic coder, each coefficient is coded according to itsown context, possibly based on which coefficient it is and possiblybased on other factors. All coefficients are coded except the zerocoefficients indicated by the bit vector described above. FIG. 8illustrates the coding of nonzero coefficients. The nonzero coefficientsfrom a list of transform coefficients 802 are coded using any coder 804.The coder outputs bits to the bitstream 806.

Other embodiments of the invention include a computer device forpracticing the method, a computer-readable medium for instructing acomputer device to practice the method of the invention, a bitstreamcreated according to a method of the present invention, and a decoderand decoder process for decoding a bitstream generated according to anembodiment of the present invention. FIG. 9 illustrates an examplemethod for decoding a bitstream. The bitstream in this example wasgenerated according to the embodiments of the invention described hereinfor generating a bitstream. The decoding method comprises decoding asingle entity, such as the bit vector, wherein the values of transformcoefficients are decoded in any fixed order (902), deconstructing thesingle entity to determine which coefficients are non-zero (904),dequantizing the non-zero transform coefficients (906), and convertingthe dequantized transform coefficients into block image data (908).

An advantage of the present invention includes enabling a mechanicaltuning of the encoder ahead of time, if desired. The mechanism is tooperate the coder on a range of typical images or video sequences toobtain typical probability distributions for all events that can becoded, then to build these typical distributions into the coder anddecoder as part of their initialization sequences. Thus no humanintervention is necessary in the tuning process.

Another advantage of this invention is that the arithmetic coderautomatically detects correlations among the various coefficientsthrough the adaptation of the bit vector probability distributions. Inaddition, using arithmetic coding guarantees that almost no bits arewasted simply because of the limitations of prefix-free codes. These andother advantages will be apparent to those of skill in the art.

Although the above description contains specific details, they shouldnot be construed as limiting the claims in any way. Other configurationsof the described embodiments of the invention are part of the scope ofthis invention. For example, the principles of the present invention maybe applied to allow coding of any related data, not just image data.There are many uses of arithmetic coding beyond image and video codingto which the fundamental principles of the present invention do apply.Accordingly, only the appended claims and their legal equivalents shoulddefine the invention, rather than any specific examples given.

I claim:
 1. A non-transitory computer-readable storage medium storing aplurality of instructions which, when executed by a processor, cause theprocessor to perform operations, the operations comprising: receiving abit vector that represents a mapping of an entire block of transformcoefficients into a one-dimensioned list of transform coefficients in afixed order, wherein the entire block of transform coefficientsrepresents a block of image data; decoding the bit vector to identifywhich transform coefficients in the entire block are non-zero; andconverting the transform coefficients into the block of image data. 2.The non-transitory computer-readable storage medium of claim 1, whereinthe decoding the bit vector is based at least in part on a context ofthe bit vector.
 3. The non-transitory computer-readable storage mediumof claim 1, the operations further comprising: decoding the non-zerotransform coefficient that is identified.
 4. The non-transitorycomputer-readable storage medium of claim 1, the operations furthercomprising: presenting an image based at least in part on the block ofimage data.
 5. The non-transitory computer-readable storage medium ofclaim 1, wherein the decoding the bit vector is accomplished using oneof: an adaptive arithmetic decoder and a non-adaptive arithmeticdecoder.
 6. The non-transitory computer-readable storage medium of claim1, wherein the bit vector comprises a single entity.
 7. An apparatus,comprising: a processor; and a non-transitory computer-readable storagemedium storing a plurality of instructions which, when executed by theprocessor, cause the processor to perform operations, the operationscomprising: receiving a bit vector that represents a mapping of anentire block of transform coefficients into a one-dimensioned list oftransform coefficients in a fixed order, wherein the entire block oftransform coefficients represents a block of image data; decoding thebit vector to identify which transform coefficients in the entire blockare non-zero; and converting the transform coefficients into the blockof image data.
 8. The apparatus of claim 7, wherein the decoding the bitvector is based at least in part on a context of the bit vector.
 9. Theapparatus of claim 7, the operations further comprising: decoding thenon-zero transform coefficient that is identified.
 10. The apparatus ofclaim 7, the operations further comprising: presenting an image based atleast in part on the block of image data.
 11. The apparatus of claim 7,wherein the decoding the bit vector is accomplished using one of: anadaptive arithmetic decoder and a non-adaptive arithmetic decoder. 12.The apparatus of claim 7, wherein the bit vector comprises a singleentity.
 13. A non-transitory computer-readable storage medium storing aplurality of instructions which, when executed by a processor, cause theprocessor to perform operations, the operations comprising: receiving abit vector that represents a mapping of an entire block of transformcoefficients into a one-dimensioned list of transform coefficients in afixed order, wherein the entire block of transform coefficientsrepresents a block of image data; decoding the bit vector to identifywhich transform coefficients in the entire block are non-zero;dequantizing the identified non-zero transform coefficients; andconverting the dequantized transform coefficients into the block ofimage data.
 14. The non-transitory computer-readable storage medium ofclaim 13, wherein the decoding the bit vector is based at least in parton a context of the bit vector.
 15. The non-transitory computer-readablestorage medium of claim 13, the operations further comprising: decodingthe identified non-zero transform coefficients.
 16. The non-transitorycomputer-readable storage medium of claim 13, the operations furthercomprising: presenting an image based at least in part on the block ofimage data.
 17. The non-transitory computer-readable storage medium ofclaim 13, wherein the decoding the bit vector is accomplished using oneof: an adaptive arithmetic decoder and a non-adaptive arithmeticdecoder.
 18. The non-transitory computer-readable storage medium ofclaim 13, wherein the bit vector comprises a single entity.
 19. Anapparatus, comprising: a processor; and a non-transitorycomputer-readable storage medium storing a plurality of instructionswhich, when executed by the processor, cause the processor to performoperations, the operations comprising: receiving a bit vector thatrepresents a mapping of an entire block of transform coefficients into aone-dimensioned list of transform coefficients in a fixed order, whereinthe entire block of transform coefficients represents a block of imagedata; decoding the bit vector to identify which transform coefficientsin the entire block are non-zero; dequantizing the identified non-zerotransform coefficients; and converting the dequantized transformcoefficients into the block of image data.
 20. The apparatus of claim19, wherein the decoding the bit vector is based at least in part on acontext of the bit vector.
 21. The apparatus of claim 19, the operationsfurther comprising: decoding the identified non-zero transformcoefficients.
 22. The apparatus of claim 19, the operations furthercomprising: presenting an image based at least in part on the block ofimage data.
 23. The apparatus of claim 19, wherein the decoding the bitvector is accomplished using one of: an adaptive arithmetic decoder anda non-adaptive arithmetic decoder.
 24. The apparatus of claim 19,wherein the bit vector comprises a single entity.
 25. A non-transitorycomputer-readable storage medium storing a plurality of instructionswhich, when executed by a processor, cause the processor to performoperations, the operations comprising: receiving a bit vector thatrepresents a mapping of an entire block of transform coefficients into aone-dimensioned list of transform coefficients in a fixed order, whereinthe entire block of transform coefficients represents a block of imagedata; decoding the bit vector for determining which transformcoefficients in the entire block are non-zero; dequantizing the non-zerotransform coefficients after determining which transform coefficients inthe entire block are non-zero; and converting the dequantized transformcoefficients into the block of image data.
 26. The non-transitorycomputer-readable storage medium of claim 25, wherein the decoding thebit vector is based at least in part on a context of the bit vector. 27.The non-transitory computer-readable storage medium of claim 25, theoperations further comprising: decoding the non-zero transformcoefficients that are identified.
 28. The non-transitorycomputer-readable storage medium of claim 25, the operations furthercomprising: presenting an image based at least in part on the block ofimage data.
 29. The non-transitory computer-readable storage medium ofclaim 25, wherein the decoding the bit vector is accomplished using oneof: an adaptive arithmetic decoder and a non-adaptive arithmeticdecoder.
 30. The non-transitory computer-readable storage medium ofclaim 25, wherein the bit vector comprises a single entity.
 31. Anapparatus, comprising: a processor; and a non-transitorycomputer-readable storage medium storing a plurality of instructionswhich, when executed by the processor, cause the processor to performoperations, the operations comprising: receiving a bit vector thatrepresents a mapping of an entire block of transform coefficients into aone-dimensioned list of transform coefficients in a fixed order, whereinthe entire block of transform coefficients represents a block of imagedata; decoding the bit vector for determining which transformcoefficients in the entire block are non-zero; dequantizing the non-zerotransform coefficients after determining which transform coefficients inthe entire block are non-zero; and converting the dequantized transformcoefficients into the block of image data.
 32. The apparatus of claim31, wherein the decoding the bit vector is based at least in part on acontext of the bit vector.
 33. The apparatus of claim 31, the operationsfurther comprising: decoding the non-zero transform coefficients thatare identified.
 34. The apparatus of claim 31, the operations furthercomprising: presenting an image based at least in part on the block ofimage data.
 35. The apparatus of claim 31, wherein the decoding the bitvector is accomplished using one of: an adaptive arithmetic decoder anda non-adaptive arithmetic decoder.
 36. The apparatus of claim 31,wherein the bit vector comprises a single entity.